Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent - #2142
Conversation
…er-turn telemetry
Adapts the ServiceNow EnterpriseOps-Gym benchmark (Apache 2.0; 8 enterprise
domains, external MCP gym servers, SQL verifiers over final DB state) to
NeMo Gym:
- resources_servers/enterpriseops_gym: per-rollout DB seeding (SQL content
cache + per-gym seed semaphores), catch-all /{tool_name} MCP proxy with
EOG-parity observations and per-tool latency capture, idempotent /verify
with guaranteed DB deletion, TTL janitor for killed rollouts, replica
pools (gym_url_pools) for horizontal MCP scale-out, and per-domain
aggregate metrics (leaderboard-style macro average).
- verifier_engine.py is a line-for-line port of the upstream engine,
preserving its quirks for score parity (verifier name-collapse where
duplicate-named verifiers overwrite; loose comparison semantics; skipped
unknown-gym verifiers), pinned by golden fixtures generated from the
original implementation. strict_verifiers=true switches the reward to
every-verifier-counts for RL shaping.
- convert_tasks.py / snapshot_tools.py convert EOG tasks (local or the
ServiceNow-AI/EnterpriseOps-Gym HF dataset) into NeMo Gym JSONL, baking
tool schemas from live tools/list snapshots with per-task gym-order
merge semantics (hybrid parity).
- benchmarks/enterpriseops: oracle public split (649 tasks) with HF
download and offline local fallbacks.
- responses_api_agents/turn_logging_agent: behaviorally identical
simple_agent variant that records per-turn telemetry (timestamps,
input/output/cached tokens, tool names) and attaches it to verify
responses; export_eval_telemetry.py emits the eval team's 21-field
per-turn JSONL schema.
Validated at full scale against the native harness on the same 649 tasks,
model, and containers: macro success 16.4-17.8% across three runs, 90-94%
per-task agreement, McNemar p>=0.21 (no detectable harness bias), and
100% identical collapsed-verifier scoring structure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Consolidates the equivalence case vs the native harness: golden-fixture unit parity, 12/12 live task parity, full-public-split single-run comparisons (McNemar p>=0.21, 100% scoring-structure agreement), and the k=5 interleaved variance experiment (6,480 rollouts): mean@5 macro 16.54+/-0.73 vs 16.76+/-0.99 (delta -0.23pp), per-task preference 67:67, and a direction-free gateway serving-path effect (+3.49pp outcome-flip excess, permutation p<0.002) attributed to Responses-vs-ChatCompletions serving rather than either harness. Includes calibration guidance (report mean@k; ~6.3% of outcomes flip on any rerun) and upstream-relevant operational findings (container fd leak, silent task drops, resume mode). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Upstream #1682 made the MCP Python SDK a core dependency, but for Gym-as-MCP-server (the inverse of this module's Gym-as-MCP-client role). Records why the SDK client is not a fit here: httpx transport (banned for high-concurrency async), session-level vs required per-call isolation headers, non-MCP REST endpoints comprising half the surface, and frozen upstream protocol version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
…ed GPU
Full oracle-split sweep (649 tasks x 2 harnesses x c in {8,16,32,64,128})
on 4xH100 TP=4 serving Nemotron 3 Nano FP8 locally, both harnesses on the
identical chat-completions endpoint. The port is 1.14-1.71x faster at
matched concurrency (largest at realistic low-c settings), needs ~4x less
client concurrency for equal throughput, and saturates the hardware at
c=64 while native never reaches the throughput floor in the tested range.
Success rates identical within noise at every level. Mechanism: GPU batch
starvation from native's per-request connections, per-task handshakes,
and sequential verifiers (GPU 98% busy both sides, client CPU idle).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Treats the 4xH100 performance sweep as an independent k=5 replication on dedicated vLLM serving (Nemotron 3 Nano FP8, both harnesses on one endpoint). Confirms the serving-path attribution via its designed falsification test (cross-harness trajectory excess 3.49 -> 1.57pp, 55% -> 11% of the noise floor), establishes formal TOST equivalence within +/-2pp at alpha=0.05 with direction-free residuals (112:106 task preference, sign p=0.735), and adds a determinism calibration for reasoning models (41% flaky tasks vs 20% non-reasoning; report mean@5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Config-only GRPO readiness validation on the 4xH100 stack: token-ID capture via return_token_id_information, curriculum selection from repeat-run sweep data, and group-mixing results (15/20 binary-mixed at k=8, mean group std 0.341). Documents the unshuffled-benchmark +limit pitfall and operational notes for long collection runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Measured train-sequence lengths (exact from v3 token IDs, band-wide via calibrated estimate): curriculum yield is 63% at a 32k cap vs 94% at 64k, which drives the shape ranking (B300/B200 single node > H200 > 2x8 H100 disaggregated > single 8x H100 > LoRA fallback). Includes trainer memory budget, step-time model, node layout, and the pre-registered proof-point run sketch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Generated by scripts/update_env_list.py; the turnlog overlay config gains a metadata block (a no-op merge over the inherited server) so the table generator can render its row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Per the environment contribution guide: data/example_rollouts.jsonl (5 pre-generated rollouts from example.jsonl against live containers) and a licensing/data-provenance section in the server README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Generated via gym dataset collate +mode=example_validation; CI's should_validate_data gate requires it alongside example.jsonl and example_rollouts.jsonl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
…h-integration-enterpriseops Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
github-actions
Bot
added
the
sla:triage-overdue
The five flagged strings in enterpriseops_gym/data/tools/drive.json are example Google Drive document IDs from the upstream EOG container's tool schemas (one is the sample spreadsheet ID from Google's own API docs), not credentials. Baseline updated with detect-secrets 1.5.0 to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
gym env test discovery only treats modules with a README.md as testable; with fail_on_total_and_test_mismatch=true the missing README failed CI's shared server-tests job (found 137 modules, tested 136). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
| @@ -0,0 +1,3436 @@ | |||
| { | |||
There was a problem hiding this comment.
can these long files be pulled from somewhere like huggingface instead of committed?
|
/claude review |
github-actions
Bot
removed
the
sla:triage-overdue
|
SHIP WITH CARE — no BLOCKERs. The verifier port and reward aggregation are the highest-risk surface here and they're the most carefully done part of the PR: EOG parity quirks (name-collapse, unknown-gym skips, loose comparisons) are documented and pinned by golden fixtures, Two non-blocking findings, both inline:
Neither blocks merge; #2 is the one I'd want resolved before this feeds a training run. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Mirror simple_agent's url_path_for_request/url_path_for_run so /ng-rollout/<id>-prefixed self-calls keep per-rollout observability correlation on downstream model calls. Adds a regression test driving the prefixed route. Addresses PR #2142 review feedback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
all([]) is True, so a task with zero scorable verifiers awarded strict reward 1.0. Guard the strict path (which feeds RL rewards); the collapsed parity path intentionally keeps upstream's all([]) semantics for leaderboard comparability, now documented in place. Addresses PR #2142 review feedback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
|
Good question. Quick context on what these are: the data/tools/*.json files are snapshots of the tool schemas advertised by EnterpriseOps-Gym's MCP
containers (512 tools across 7 domains). They can't come from HuggingFace as-is because the upstream HF dataset only contains tasks; the schemas exist
only inside the running Docker containers, and they're needed at prepare.py time to build task rows. Pinned copies are also what guarantees byte-level
prompt parity with the upstream harness (see PARITY.md).
I agree they're bulky. Two options:
1.
Host them on HF (a small companion dataset, or the existing dataset repo if ServiceNow will take them, which would need upstreaming) and have prepare.py download them, keeping only a checksum in-repo. Downside: prepare gains a network dependency for data that changes only when upstream ships new containers.
2.
Keep them committed as the reproducibility anchor. They're versioned exactly like golden test fixtures.
My mild preference is (2) for reproducibility, but (1) works if repo size is the priority.
From: Christian Munley ***@***.***>
Date: Monday, July 27, 2026 at 10:13 AM
To: NVIDIA-NeMo/Gym ***@***.***>
Cc: Marc Cuevas ***@***.***>; Author ***@***.***>
Subject: Re: [NVIDIA-NeMo/Gym] Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent (PR #2142)
@cmunley1 commented on this pull request.
________________________________
In resources_servers/enterpriseops_gym/data/tools/calendar.json<#2142 (comment)>:
@@ -0,0 +1,3436 @@
+{
can these long files be pulled from somewhere like huggingface instead of committed?
—
Reply to this email directly, view it on GitHub<#2142?email_source=notifications&email_token=B55WDQREOH2VD5KOXZ7ALYD5G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4789624130>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/B55WDQVTNXPZ77HYPHRFJJD5G6ERPAVCNFSNUABGKJSXA33TNF2G64TZHMYTANBUGU2TIOBUGA5US43TOVSTWNBZG4ZTENRWHEYTTILWAI>.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/B55WDQVOGQBHPOBD2ZQZR735G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/B55WDQR4JTS3JH3KU26HLPL5G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today!
You are receiving this because you authored the thread.
|
…h-integration-enterpriseops Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
github-actions
Bot
added
the
sla:review-overdue
github-actions
Bot
removed
the
sla:review-overdue
github-actions
Bot
added
the
sla:review-overdue
|
Could we do option 1 @mcuevas-nvidia ? |
|
/claude review |
| } | ||
| ], | ||
| "results": { | ||
| "resources_servers/enterpriseops_gym/data/tools/drive.json": [ |
There was a problem hiding this comment.
could you use inline # pragma: allowlist secret instead of global config?
There was a problem hiding this comment.
we might wanna remove images for keeping repo slim as we add many envs
There was a problem hiding this comment.
its fine if we leave this, its not too big.
| @@ -0,0 +1,34 @@ | |||
| # EnterpriseOps-Gym with the per-turn-telemetry agent (eval-team logging schema). | |||
| # Adds a turn_logging_agent alongside the standard stack; collect with | |||
| # ng_collect_rollouts +agent_name=enterpriseops_gym_turn_logging_agent ... | |||
There was a problem hiding this comment.
could u update this to new cli
| --output resources_servers/enterpriseops_gym/data/csm_revised.jsonl | ||
|
|
||
| # Run servers + collect rollouts | ||
| ng_run "+config_paths=[resources_servers/enterpriseops_gym/configs/enterpriseops_gym.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" |
There was a problem hiding this comment.
new cli also would be good
There was a problem hiding this comment.
yes we should update to the latest commands
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| """SimpleAgent with per-turn telemetry. |
There was a problem hiding this comment.
Just a comment: I wonder if this should go directly into simple agent or we see if the model call capture recent work is sufficient. I know there are some other efforts on observability.
| # `num_verifiers_scored` exposes the condition to callers. | ||
| total_verifiers = len(verification_results) | ||
| passed_verifiers = sum(1 for v in verification_results.values() if v.get("passed", False)) | ||
| overall_success = all(v["passed"] for v in verification_results.values()) |
There was a problem hiding this comment.
NOTE — silent score inflation on config error (default parity mode).
WHAT: With strict_verifiers: false (the default, per enterpriseops_gym.yaml), if every verifier is skipped for an unknown gym_name, verification_results is empty → overall_success = all([]) == True → reward = 1.0, while verifier_pass_rate is 0.0.
BLAST RADIUS: A fat-fingered gym_servers_config.mcp_server_name (so every verifier's gym_name misses) makes every rollout in that eval score a false pass, inflating the report. You've correctly documented the deliberate empty-set parity quirk (PARITY.md §1) and strict mode fixes it — the concern is specifically the config-typo path, which looks identical to a legitimate all-pass.
FIX (optional, defense-in-depth): the data is already there — num_verifiers_scored/num_verifiers_defined are surfaced. Consider a startup/seed-time warn (or an eval-side check) when num_verifiers_scored == 0 for a nonzero num_verifiers_defined, so a mass-skip surfaces instead of reading as 100%. Author's call — not a merge blocker.
|
SHIP — no blocking reliability concerns. Reviewed the full change (50 files): the What I checked hardest, and what held up:
One inline NOTE (defense-in-depth, author's call): in default parity mode, a |
Resolves the README environment table by regenerating it with scripts/update_env_list.py, and takes main's .secrets.baseline (the enterpriseops entries are removed in a follow-up commit that deletes the file they reference). Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Replaces the deprecated ng_* aliases with their `gym` equivalents per fern/versions/latest/pages/reference/cli-commands.mdx: ng_run -> gym env start ng_collect_rollouts -> gym eval run --no-serve ng_prepare_benchmark-> gym eval prepare and converts the Hydra +key=value overrides to the corresponding flags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
…ing them The seven per-domain tools/list snapshots are 30,724 lines of generated JSON -- 86% of this PR's added lines -- and are build-time inputs only: convert_tasks bakes them into dataset rows at prepare time and nothing reads them at run time. snapshot_tools.py can re-capture them from the upstream containers at any time. Follows the conversational_tool_use_simulation precedent: a new prepare.py fetches nvidia/NeMo-Gym-EnterpriseOps-Assets at a pinned revision, validates it against a (file_count, tree_sha256) pin, and materializes data/tools/, which is now gitignored. ensure_tool_snapshots() is idempotent and runs before the benchmark opens its output file, so a download failure cannot truncate an existing benchmark JSONL. NEMO_GYM_EOG_TOOLS_DIR skips the download for air-gapped machines, validated against the same pin. Reverts the .secrets.baseline entries, which only existed for drive.json -- that closes the inline-pragma request too, since there is nothing left to allowlist. Also drops csm_revised.jsonl and itsm_revised.jsonl (405 KB): both are output of the documented convert_tasks command, and neither is referenced by any code, config, or test. DEFAULT_REVISION is a placeholder until the dataset repo is published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
With strict_verifiers false (the default), a gym_servers_config typo means no verifier matches a live gym, verification_results is empty, and the parity path scores all([]) as a pass. That is deliberate -- it is upstream EOG behavior and PARITY.md pins it -- but in aggregate metrics it is indistinguishable from a genuine 100% pass, so a mass skip can silently inflate a whole eval. Log a warning naming the referenced gym_name(s) and the session's actual gyms when verifiers were defined but none were scored. Scoring is unchanged; this is observability only. The warning fires only on a total skip, not a partial one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
nvidia/NeMo-Gym-EnterpriseOps-Assets is published; replace the placeholder with its head SHA. Verified with the committed defaults against a cold HF cache: download, filename and checksum validation, and materialization all succeed, and the six committed rows still reproduce byte-identical tool schemas from the hosted snapshots. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
The dataset is now public and ungated. Re-pin to the revision carrying the corrected NOTICE, which describes provenance rather than asserting upstream copyright ownership (upstream ships no NOTICE file, so there are no attribution notices to propagate under Apache-2.0 section 4(d)). Verified anonymously with no token and a cold cache: download, validation, and materialization all succeed, so external contributors and CI can prepare. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
Running the full 649-task oracle conversion surfaces two tasks whose selected_tools do not resolve against the container tool surface. Record the cause, the evidence that it is not snapshot staleness, and the bound on the headline metric. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
#2827 (2026-08-28) made every resources server ship a task_data.py describing its dataset rows; it landed after this PR branched, so the merge of main left two repo tests failing. Rows nest everything in an untyped verifier_metadata bucket, so the schema is flat with legacy_location annotations. Every field is Optional because the server reads the bucket only via .get(...) or <default> and never 422s on its contents. Shapes derived from all 655 committed and generated rows: 744 gym_servers_config entries and 3,532 verifiers, with user_info the one variable field (absent from 44 entries, str or dict where present). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Marc Cuevas <mcuevas@nvidia.com>
|
Addressed all PR comments. Ready for re-review. The seven Approach follows the
A fully air-gapped machine now needs Also in this push:
Verified: the download → checksum → materialize path end-to-end against the real Hub Status: the dataset is published and public at |
3 participants
Summary
This PR integrates ServiceNow's EnterpriseOps-Gym (EOG) into NeMo Gym: a 649-task benchmark of stateful, multi-step enterprise tool use across 8 domains (calendar, CSM, drive, email, HR, ITSM, teams, and cross-domain hybrid), where an agent operates 512 tools against live MCP servers backed by SQL databases and is scored on final database state. The integration is eval-complete and RL-ready (token-ID capture, fractional reward mode, and a validated GRPO rollout-collection recipe), with no changes to NeMo Gym core.
What's included (46 files)
resources_servers/enterpriseops_gym//{tool_name}route, pooled aiohttp, per-sessionx-database-id), runs verifiers concurrently, cleans up idempotently (TTL janitor + delete-on-verify). Supports replica pools (gym_url_pools), per-domain metrics, per-tool latency capture, and astrict_verifiersfractional-reward mode for RL..../verifier_engine.py.../convert_tasks.py+benchmarks/enterpriseops/ServiceNow-AI/EnterpriseOps-Gym) to Responses-API task rows atprepare.pytime; benchmark registered withprompt_config: null(pre-baked rows). Dataset files are not committed.responses_api_agents/turn_logging_agent/SimpleAgentsubclass with an identical loop that records per-turn timestamps, durations, input/output/cached/reasoning tokens, and tool names, attachingturnsto the verify response. Generic — not EOG-specific.tests/stub_gym.py) plus golden parity fixtures. No containers or network needed to run CI.PARITY.md,PERF.md,RLPILOT.mdVerification logic
Each task carries
verifier_metadatawith a list of verifiers.database_stateverifiersrun a SQL query against the session's final database state (via the gym containers'
/api/sql-runner), extract a value, and compare it to the expected value using EOG'scomparison semantics;
response_checkverifiers score the agent's final message with anLLM judge (defaults to the policy model, temperature pinned to 0.0, matching EOG). The
public oracle split is 100%
database_state(3,496/3,496 verifiers), so scoring there isfully deterministic given a final DB state. Reward = 1.0 iff all (name-collapsed) verifiers
pass, matching the upstream leaderboard;
strict_verifiers: trueswitches toevery-verifier-counts and a fractional
strict_pass_ratefor RL shaping.Scoring fidelity (PARITY.md)
The port is validated bug-for-bug against the upstream harness, preserving its quirks (verifier name-collapse, unknown-gym skips, loose comparison semantics) for leaderboard comparability:
Performance (PERF.md)
Scale-tested end-to-end at five client concurrencies (c = 8, 16, 32, 64, 128) with a full
649-task pass per level per harness (10 passes, 4×H100 vLLM, identical endpoint): the port
completed 649/649 tasks at every level with zero retries, with success rates flat
across levels. At matched concurrency the port is 1.14–1.71× faster (largest at low
concurrency, nearest the native harness's documented defaults). The gap narrows by design:
both harnesses converge toward the same GPU throughput floor — and reaching it is the key
result. The port saturates the hardware at c=64; the native harness never reaches the
floor in the tested range and needs ~4× the client concurrency for equal throughput.
Net cost: a full-split eval is 1h27m of 4×H100 time (port) vs 4h40m at native's documented
settings — 3.2× GPU-hours. Mechanism: pooled connections, persistent MCP sessions, and
concurrent verifiers keep vLLM's continuous batch fed. (Since the gap is client dead time
relative to GPU service time, it is expected to widen on faster serving hardware, where
saturation demands even more effective concurrency.)
RL readiness (RLPILOT.md)
A config-only pilot (zero code changes) validated GRPO rollout collection end-to-end: 100% token-ID/logprob coverage via
return_token_id_information, 15/20 task groups with mixed binary reward at k=8 (mean within-group std 0.341) on a curriculum selected from repeat-run data, plus a measured deployment-sizing guide (sequence-length distribution, memory budget, recommended node shapes).How to run
Validation on this exact branch state
mainruff check+ruff format --checkclean; README environment table regenerated viascripts/update_env_list.pyng_rundry runContribution-guide compliance
Mapped to the environment /
benchmark guides:
app.py,configs/*.yaml(validdomain: agent),tests/test_app.py(42 tests),data/example.jsonl(5 tasks),data/example_rollouts.jsonl(5 pre-generated rollouts against live containers; notethese samples are CSM-domain, the benchmark's hardest — near-zero rewards on them are
expected and consistent with the full-split CSM rate of ~4%),
requirements.txt,README.mdwith licensing information.thinking model (Nemotron 3 Nano, reasoning on: 22–25%), i.e. the instruct+thinking
mixture the guide asks for. Scores sit inside the official leaderboard's published range
(Qwen3-4B 13.6% … GPT-5-Mini 22.0%) with a coherent domain pattern (email easiest, CSM
hardest, matching the leaderboard). This benchmark is legitimately hard — no public model
reaches 30%.
Calibration guidance for users is included in PARITY.md.
(documented in the three reports).
number (leaderboard models weren't available on our serving), we ran the original EOG
harness side-by-side on identical models, containers, and endpoints: per-task agreement
with symmetric disagreements (McNemar p = 0.90) and formal TOST equivalence within ±2 pp.
This isolates harness fidelity even more directly; happy to additionally run a listed
leaderboard model if reviewers want the published-number check.
(resources server + tests → benchmark registration → turn-logging agent → reports) if
reviewers prefer; presented as one PR first since the layers are tightly coupled by the
parity evidence.
Design decisions & notes for reviewers
mcp_client.pydocstring): the SDK's transport is httpx-based (banned for async here), EOG needs session-level and per-callx-database-idheaders, and half the surface is non-MCP REST (/api/seed-databaseetc.).strict_verifiers: trueopts into every-verifier-counts scoring for RL.turn_logging_agentis separable — it's a general-purpose agent; happy to split it into its own PR if preferred.verified: falseper convention for new resources servers.Data provenance & licensing
EnterpriseOps-Gym is Apache 2.0 (code) with a public HF dataset (
ServiceNow-AI/EnterpriseOps-Gym). This PR commits only: tool-schema snapshots captured from the public EOG containers (7 JSON files, 512 tools), 13 sample tasks derived from the EOG repo's task files, 5 example rollouts generated against live containers, one synthetic hybrid task hand-authored for tests (written with LLM assistance against live container schemas — disclosed per the synthetic-data guideline), and golden verifier fixtures generated by running the EOG engine. The full benchmark dataset is downloaded atprepare.pytime and gitignored.